2021.1 (latest)

Itential Automation Platform

On this page:

Event Deduplication

Overview

In order for the event deduplication feature to work, all involved RabbitMQ instances must be configured beforehand, and the feature itself must be enabled globally through the system profile.

The feature operates using a third-party plugin to RabbitMQ, and has the ability to be configured on a service-by-service basis which event message properties should be used to determine when a message should be dropped.

When event deduplication settings change, IAP must be restarted for those changes to be propagated to RabbitMQ. If IAP is deployed in an HA cluster, all instances of IAP must be restarted in order for the event system to operate. Restarting only one instance will result in undefined behavior due to the fact that all other servers will be running with an outdated profile.

Due to the nature of RabbitMQ, propagating any configuration changes involves deleting and recreating the iap_event_exchange exchange and iap_event queue, since RabbitMQ resource properties may not be updated once created. To handle cases where there are unprocessed messages left in the event queue when a recreation occurs, IAP provides the ability to configure an event message preservation strategy. This feature utilizes the rabbitmq_shovel plugin to move all transient messages to a temporary queue while the iap_event_exchange exchange iap_event queue are recreated with new properties. This preservation strategy helps to prevent message loss.

Purpose

The event deduplication feature is designed to handle situations where the same event might occur from multiple sources, especially in the context of highly-available architecture. If multiple downstream resources can report the same event, or if multiple IAP instances are listening to events from the same downstream resource, multiple event messages may be broadcast in the IAP event system for the same downstream occurrence. In this case, actions listening for those events will execute multiple times. In most cases, however, the desired behavior is to respond only once for each distinct occurrence.

For example, event deduplication may be required if an instance of NSO is observed by a cluster of IAP servers. In this scenario, each IAP server will maintain its own individual connection to the NSO server. NETCONF events produced by the NSO server will be observed by all running instances of IAP, resulting in multiple messages being generated in the IAP event system for a single NETCONF event. In this case, IAP event deduplication may be configured to ensure that only one message is emitted to the event bus so that there is only one response to the event from the system as a whole.

Profile Activation

To set an active IAP profile:

  1. Verify installation of RabbitMQ plugins. Please see the section on RabbitMQ Installation in the IAP Admin Guide.
  2. Configure IAP profile.
  3. Configure service config of each event producer which requires event deduplication.

Profile Configuration

In order for IAP to use the installed RabbitMQ plugins, the event deduplication feature must be enabled in the rabbitmq section of the active profile. See below for an example configuration setting that can be added to the IAP Profile:

Example

The following is a sample profile configuration with exhaustive defaults.

{
  ...,
  "rabbitmq": {
    ...,
    "adminPort": 15672,
    "eventDeduplication": {
      "active" : false,
      "cacheSize" : 1000,
      "cacheTtl" : 100,
      "cachePersistence" : "memory",
      "messagePreservation" : {
        "strategy" : "automatic",
        "tempQueueName" : "iap_event__TMP__",
        "shovelName" : "preserve_iap_events",
        "shovelAckMode" : "on-confirm",
        "shovelAddHeaders" : false,
        "shovelPrefetchCount" : 1000,
        "shovelReconnectDelay" : 5000,
        "maxTransferWait" : 5000,
        "transferWaitPollIncrement" : 100
      }
    }
  }
}

adminPort

The adminPort property specifies the port the RabbitMQ Management API is exposed on. This API is used to manage event deduplication configuration and must be exposed and specified in your profile settings in order for event deduplication to operate correctly.

eventDeduplication

Use this table to set eventDeduplication properties for the active profile.

Property Type Default Description
active Boolean False Turns event deduplication functionality on or off at the system level.
cacheSize Integer 1000 Denotes the maximum number of messages that will be cached when deduplicating events.
cacheTtl Integer 100 Denotes the time in milliseconds the unique identifier for a message will be cached after it is published.
cachePersistence Enum
("memory",
"disk")		 Memory Denotes which type of memory to use when caching message identifiers.
messagePreservation Object Enabled Container for settings related to preservation of transient messages left from a previously running IAP cluster. See section below.

messagePreservation

It is theoretically possible to have unprocessed messages left in the event queue if a catastrophic error occurs (lost network connectivity to RabbitMQ or an event is published just as IAP restarts). For this reason, IAP offers configurable options for preserving unprocessed event messages. The default options for this group of settings are enabled by default for safety.

Property Type Default Description
strategy Enum
("automatic",
"manual",
"off")		 Automatic Strategy for handling transient messages. Enum options are:
automatic will move residual messages to a temporary queue while the event queue is recreated, and then restore them afterwards.
manual will abort IAP startup when messages exist in the event queue so that an engineer can rectify the situation.
off will update event deduplication settings regardless of transient messages, destroying them in the process. The off option is not recommended, but may be necessary in some situations at the discretion of the administrator.
tempQueueName String iap_event__TMP__ Name of temporary container queue. Used to hold transient messages while recreating the iap_event queue with new settings.
shovelName String preserve_iap_events Name of the temporary queue used to move messages to and from the temporary container queue.
shovelAckMode Enum
("on-confirm",
"on-publish",
"no-ack")		 on-confirm Refer to the official shovel plugin documentation for detailed information on the ack-mode setting.
shovelAddHeaders Boolean False Denotes whether or not the x-shovelled header should be set on messages processed by the shovel.
shovelPrefetchCount Integer 1000 Number of messages the shovel will transfer at a given time.
shovelReconnectDelay Integer 5000 Duration in milliseconds to wait if the broker becomes disconnected.
maxTransferWait Integer 5000 Duration in milliseconds to wait for message transfer both to and from the temporary queue.
transferWaitPollIncrement Integer 100 Increment in milliseconds used when polling the message count in the queue being drained by the transfer shovel.

Service Configuration

Once the active IAP profile is configured to enable event deduplication, each service for which events are to be deduplicated must be configured. The most important settings shown below are active and uniqueProps, which determine the on/off status of the feature, as well as the set of message properties that will be used to determine which messages are duplicates. The uniqueProps allow you to handpick the fields that will be used to evaluate if two events are to be regarded as duplicates. For example, if the values of the three fields specified below ("/value/device", "/value/event", "/value/timestamp") are the same between two events, the event is considered as a duplicate. If any of these are different, then the events are not deduplicated.

Example

The following is a sample service config for deduplication settings.

{
  ...,
  "eventDeduplication": {
    "active" : true,
    "cacheTtl" : 100,
    "algorithm" : "md5",
    "uniqueProps" : [
      "/value/device",
      "/value/event",
      "/value/timestamp"
    ]
  }
}

eventDeduplication

Use this table to set eventDeduplication properties for the service config.

Property Type Default Description
active Boolean False Turns event deduplication on or off at the component level.
cacheTtl Integer 100 Overrides the global message identifier time-to-live setting for this service.
algorithm String sha512 Algorithm used to hash event messages when producing a message identifier. See below for more info.
uniqueProps Enum
("array",
"object",
null)		 Null
(Use all properties.)		 Denotes which message fields will be used when producing a message identifier. This setting may be used to exclude fields which may vary slightly between event producers or do not pertain to the uniqueness of the message. See below for more detail.

algorithm

The algorithm setting is unlikely to be required, but in the case that it is, available options are discoverable via openssl list -digest-algorithms, or openssl list-message-digest-algorithms on older systems. The library used to create a message digest uses the system's openssl utility to produce a result.

uniqueProps

The uniqueProps setting is used to tell the deduplication system which fields define a unique message. The fields specified in this property are used to extract a subset of the message which is then hashed to provide a unique value to the deduplication system.

For example, if the uniqueProps property specifies the fields device, event and timestamp, the following messages will be considered as redundant to each other, and two of them will be dropped while the first is emitted through the event system.

{
  "device": "ios1",
  "event": "CRASH",
  "timestamp": 1592329817,
  "remediation": "REBOOT"
}

{
  "device": "ios1",
  "event": "CRASH",
  "timestamp": 1592329817,
}

{
  "device": "ios1",
  "event": "CRASH",
  "timestamp": 1592329817,
  "region": "us-east-1"
}

When using inclusive configuration (shown below as JSON Pointer Array and Inclusive Mask Object), it is mandatory that uniqueProps be specified explicitly. In the event that none of the specified fields are found in a message, it is considered unique and subsequent messages like it will not be deduplicated.

JSON Pointer Array

When uniqueProps is an array, it must contain a list of JSON pointer expressions. These expressions tell IAP which properties should be used when comparing messages, forming an inclusive list.

[
  "/value/device",
  "/value/event",
  "/value/timestamp"
]

Inclusive Mask Object

The uniqueProps setting may also be specified as an object with true or false as its leaf values (example below). In this form, uniqueProps acts as a 'mask' object. Keep in mind, true and false leaf values may not be mixed, as this causes ambiguity in the mask object in some situations.

The example below is equivalent to the JSON Pointer Array example above.

{
  "value": {
    "device": true,
    "event": true,
    "timestamp": true
  }
}

The inverse of the above is also possible for use in case the messages from an event source are deduplicated based on all but a known set of properties. For example, if the message in question also contains properties for region and severity, the following mask object would be equivalent to both of the above examples:

{
  "value": {
    "region": false,
    "severity": false
  }
}